Layer-Wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition

The variety and complexity of accents pose a huge challenge to robust Automatic Speech Recognition (ASR). Some previous work has attempted address such problems, however most the current approaches either require prior knowledge about target accent, or cannot handle unseen accent-unspecific standard speech. In this work, we aim improve multi-accent speech recognition in end-to-end (E2E) framework with novel layer-wise adaptation architecture. Firstly, propose deep accent representation learning architecture obtain accurate embedding, some advanced schemes are designed further boost quality embeddings, including phone posteriorgram (PPG) feature, TTS based data augmentation training stage, test-time multi-embedding fusion testing stage. Then, embeddings is developed for fast ASR, two types adapter layers designed, gated layer multi-basis layer. Compared usual two-pass adaptation, these injected between ASR encoder encode information flexibly, perform adaption on corresponding accent. experiments Accent AESRC corpus show that proposed can capture knowledge, get high performance classification. new embedding outperforms other traditional methods, obtains consistent $\sim$15% relative word error rate (WER) reduction all kinds scenarios, seen accents,